As UX researchers, we often collect both survey feedback and behavioral data during usability testing. Ideally, those measures tell a consistent story. Sometimes they don't.
While reviewing the results of three usability studies conducted on a payroll and human resources platform, I noticed an interesting pattern. Survey responses suggested that users were generally satisfied with the application. However, task performance data and behavioral observations revealed significant usability issues.
Task success rates ranged from 13% to 100% depending on the workflow being tested. Users frequently struggled to locate important functionality, became confused about where tasks should be completed, and relied on trial-and-error behavior to accomplish their goals. Some users were unable to complete assigned tasks at all.
The disconnect became even more apparent when I compared survey ratings with task performance across the three studies. In the first study, 100% of participants reported overall satisfaction with the application, yet users struggled with terminology, hidden functionality, unclear save states, and workflow navigation. In the second study, 88% of participants reported overall satisfaction, even though some workflows achieved success rates as low as 60% and participants described relying on trial-and-error behavior to complete tasks.
By the third study, the behavioral data painted an even clearer picture. Users encountered hidden controls, horizontal scrolling, multiple scroll bars, and confusing workflow interactions. Some payroll processing tasks achieved success rates as low as 13%, despite participants generally expressing positive opinions about the application.
One finding was particularly revealing. In a task that relied on color-only indicators to communicate important information, every participant failed to complete the task successfully. Despite the complete task failure, overall survey ratings remained generally positive.
If I had looked only at the survey results, I would have concluded that users were largely satisfied with the application. By observing user behavior and measuring task success, a very different picture emerged.
This experience reinforced an important lesson: surveys tell us what users think about an experience, while behavioral data shows us what actually happened during the experience. Both measures are valuable, but neither provides the full story on its own.
Open to contract UX research opportunities.
Email: theresaw@columbus.rr.com
LinkedIn: theresa-wilkinson